Skip to content

Comments

Faster spatialdata import time#1075

Open
LucaMarconato wants to merge 6 commits intomainfrom
faster-import
Open

Faster spatialdata import time#1075
LucaMarconato wants to merge 6 commits intomainfrom
faster-import

Conversation

@LucaMarconato
Copy link
Member

@LucaMarconato LucaMarconato commented Feb 18, 2026

Gives a shot at #456. I'll close it, but in the future we could explore further optimizations (see the profimp report below).

@ivirshup @flying-sheep I'll tag you as reviewers since you did similar work in scverse/scanpy#756.

This PR:

  • makes the import of spatialdata and submodules lazy
    • also add tests to ensure that the lazy-import machinery remains in sync
  • reduces the burden of importing some heavier packages used sporadically, by making those imports local
  • improves the asv benchmark
  • addes perfimp to the dev requirements

Results:

| Change   | Before [3d9dee3e] <main>   | After [ed9f72f4] <faster-import>   |   Ratio | Benchmark (Parameter)                            |
|----------|----------------------------|------------------------------------|---------|--------------------------------------------------|
| -        | 1.04±0.02s                 | 828±5ms                            |    0.79 | benchmark_imports.timeraw_import_read_zarr       |
| -        | 1.06±0.1s                  | 780±7ms                            |    0.74 | benchmark_imports.timeraw_import_SpatialData     |
| -        | 1.09±0.05s                 | 768±7ms                            |    0.7  | benchmark_imports.timeraw_import_models_elements |
| -        | 1.05±0.01s                 | 583±5ms                            |    0.56 | benchmark_imports.timeraw_import_transformations |
| -        | 1.05±0.05s                 | 20.3±0.1ms                         |    0.02 | benchmark_imports.timeraw_import_spatialdata     |

Output from profimp

Command profimp "from spatialdata import read_zarr" --html > profimp.html && open profimp.html
image

@codecov
Copy link

codecov bot commented Feb 18, 2026

Codecov Report

❌ Patch coverage is 96.66667% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.95%. Comparing base (3d9dee3) to head (0dce65b).
⚠️ Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
src/spatialdata/__init__.py 96.00% 1 Missing ⚠️
src/spatialdata/__main__.py 0.00% 1 Missing ⚠️
src/spatialdata/io/__init__.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1075      +/-   ##
==========================================
+ Coverage   91.93%   91.95%   +0.01%     
==========================================
  Files          51       51              
  Lines        7664     7715      +51     
==========================================
+ Hits         7046     7094      +48     
- Misses        618      621       +3     
Files with missing lines Coverage Δ
src/spatialdata/_core/_deepcopy.py 98.41% <100.00%> (+0.02%) ⬆️
src/spatialdata/_core/_elements.py 92.22% <100.00%> (+0.08%) ⬆️
src/spatialdata/_core/_utils.py 100.00% <100.00%> (ø)
src/spatialdata/_core/centroids.py 100.00% <100.00%> (ø)
src/spatialdata/_core/concatenate.py 92.53% <100.00%> (+0.05%) ⬆️
src/spatialdata/_core/data_extent.py 97.90% <100.00%> (+0.01%) ⬆️
src/spatialdata/_core/operations/aggregate.py 94.38% <100.00%> (+0.06%) ⬆️
src/spatialdata/_core/operations/map.py 98.80% <100.00%> (+0.01%) ⬆️
src/spatialdata/_core/operations/rasterize.py 90.26% <100.00%> (+0.09%) ⬆️
src/spatialdata/_core/operations/rasterize_bins.py 93.79% <100.00%> (ø)
... and 34 more
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@LucaMarconato LucaMarconato marked this pull request as ready for review February 18, 2026 17:25
@LucaMarconato
Copy link
Member Author

LucaMarconato commented Feb 18, 2026

I've done a similar PR in spatialdata-io: scverse/spatialdata-io#373 (but the diff is messy because it depends on another PR that is still unmerged).

Comment on lines +186 to +187
[tool.ruff.lint.isort]
required-imports = ["from __future__ import annotations"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added this. It makes it so the UP lints will always flag when you’re just using an import in an annotation, and therefore will automatically skip such imports at runtime by sticking them into if TYPE_CHECKING: blocks.

Copy link
Member

@flying-sheep flying-sheep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

In case you want to do everything we do in scanpy, we also do this to enforce that a list of slow-to-import packages aren’t accidentally imported:

  1. define a pytest plugin that isn’t part of the scanpy module (testing.scanpy._pytest or a conftest.py in the repository root both work, the former has to be activated using tool.pytest.addopts = ["-p", "testing.scanpy._pytest"])
  2. import scanpy in there and record sys.modules before and after
  3. check the diff against the list of to-be-lazy-loaded modules.

unfortunately that approach means we can’t use our own warnings in the tool.pytest.filterwarnings setting, but I can’t think of another way to do it except for importing it in a subprocess.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants